Skip to content

fix: memory over allocation due to last partial block group - #832

Open
siddh34 wants to merge 1 commit into
apple:mainfrom
siddh34:fix/memoryOverAllocation
Open

fix: memory over allocation due to last partial block group#832
siddh34 wants to merge 1 commit into
apple:mainfrom
siddh34:fix/memoryOverAllocation

Conversation

@siddh34

@siddh34 siddh34 commented Aug 6, 2026

Copy link
Copy Markdown

Description

This resolves issue #647

This is fix for memory over allocation due to last partial block

I have explicitly laid out metadata for extra/last groups

Modifications:

Packed-region math is computation starts at line 718

function to calculate blocks in last group at line 58

while loop here didn't made any sense so replaced it with if check line 914

@dkovba dkovba left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the contribution!
Below are the test results across the size/content matrix, cross-checked with e2fsck:

Requested Size: 32KiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 64MiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 128MiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 128MiB+4KiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 130MiB+8KiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 160MiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 256MiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 1GiB
Content: none
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 4GiB
Content: none
Size: FAIL
allocatedMiB <= ceilingMiB = false; allocatedMiB = 65; ceilingMiB = 32
e2fsck: PASS
----------------------------------------
Requested Size: 63x128MiB
Content: none
Size: FAIL
allocatedMiB <= ceilingMiB = false; allocatedMiB = 127; ceilingMiB = 32
e2fsck: PASS
----------------------------------------
Requested Size: 63x128MiB+4KiB
Content: none
Size: FAIL
allocatedMiB <= ceilingMiB = false; allocatedMiB = 129; ceilingMiB = 32
e2fsck: FAIL rc=4
Block bitmap differences:  +(32768--32930)
Fix? no
/imgs/63x128MiB+4KiB.img: ********** WARNING: Filesystem still has errors **********
/imgs/63x128MiB+4KiB.img: 11/524288 files (0.0% non-contiguous), 32737/2064385 blocks
----------------------------------------
Requested Size: 8GiB
Content: none
Size: FAIL
allocatedMiB <= ceilingMiB = false; allocatedMiB = 129; ceilingMiB = 32
e2fsck: FAIL rc=4
Block bitmap differences:  +(32768--32930)
Fix? no
/imgs/8GiB.img: ********** WARNING: Filesystem still has errors **********
/imgs/8GiB.img: 11/524288 files (0.0% non-contiguous), 32737/2097152 blocks
----------------------------------------
Requested Size: 16GiB
Content: none
Size: FAIL
allocatedMiB <= ceilingMiB = false; allocatedMiB = 258; ceilingMiB = 32
e2fsck: FAIL rc=4
Block bitmap differences:  +(32768--65826)
Fix? no
/imgs/16GiB.img: ********** WARNING: Filesystem still has errors **********
/imgs/16GiB.img: 11/1048576 files (0.0% non-contiguous), 32737/4194304 blocks
----------------------------------------
Requested Size: 160MiB
Content: 10MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 128MiB
Content: 50MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 160MiB
Content: 120MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 160MiB
Content: 124MiB
Size: PASS
e2fsck: FAIL rc=4
Block bitmap differences:  +(32768--32806)
Fix? no
/imgs/160MiB_124MiB-content.img: ********** WARNING: Filesystem still has errors **********
/imgs/160MiB_124MiB-content.img: 135/16384 files (0.0% non-contiguous), 32737/40960 blocks
----------------------------------------
Requested Size: 160MiB
Content: 126MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 256MiB
Content: 130MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 1GiB
Content: 200MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 300MiB
Content: 260MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 4GiB
Content: 260MiB
Size: PASS
e2fsck: PASS
----------------------------------------
Requested Size: 63x128MiB
Content: 500MiB
Size: PASS
e2fsck: FAIL rc=4
Block bitmap differences:  +(131072--160417)
Fix? no
/imgs/63x128MiB_500MiB-content.img: ********** WARNING: Filesystem still has errors **********
/imgs/63x128MiB_500MiB-content.img: 511/516096 files (0.0% non-contiguous), 131041/2064384 blocks
----------------------------------------
Requested Size: 8GiB
Content: 300MiB
Size: PASS
e2fsck: FAIL rc=4
Block bitmap differences:  +(98304--109730)
Fix? no
/imgs/8GiB_300MiB-content.img: ********** WARNING: Filesystem still has errors **********
/imgs/8GiB_300MiB-content.img: 311/524288 files (0.0% non-contiguous), 98273/2097152 blocks
----------------------------------------
Requested Size: 16GiB
Content: 1000MiB
Size: PASS
e2fsck: FAIL rc=4
Block bitmap differences:  +(262144--321828)
Fix? no
/imgs/16GiB_1000MiB-content.img: ********** WARNING: Filesystem still has errors **********
/imgs/16GiB_1000MiB-content.img: 1011/1048576 files (0.0% non-contiguous), 262113/4194304 blocks
----------------------------------------

@siddh34

siddh34 commented Aug 7, 2026

Copy link
Copy Markdown
Author

@dkovba thanks for the tests I will update my pr in few days and will include these failed tests in it too!

@siddh34
siddh34 force-pushed the fix/memoryOverAllocation branch from 98555ce to 16a7a7b Compare August 11, 2026 03:07
@siddh34
siddh34 requested a review from dkovba August 11, 2026 03:36
@siddh34

siddh34 commented Aug 11, 2026

Copy link
Copy Markdown
Author

Updates after issues discovered in new test cases as given by @dkovba

  1. Reduced the allocated size by

defining minimum inodes per group at line 41

This tries multiple candidate values for inodesPerGroup line 1093 it basically means smaller inode tables less metadata overhead

  1. Bitmap fixes as follows

Now ext4 block bitmaps are per-group. Changes can be found at line 879

basically marking the packed metadata blocks that physically live inside a group as allocated in that group’s block bitmap

also tested with e2fsck

CASE: 32KiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/32KiB_none.img
  requested: 32768 bytes
  content: 0 bytes
  allocatedMiB: 0
  e2fsck rc: 0
CASE: 64MiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/64MiB_none.img
  requested: 67108864 bytes
  content: 0 bytes
  allocatedMiB: 0
  e2fsck rc: 0
CASE: 128MiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/128MiB_none.img
  requested: 134217728 bytes
  content: 0 bytes
  allocatedMiB: 0
  e2fsck rc: 0
CASE: 128MiB+4KiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/128MiB+4KiB_none.img
  requested: 134221824 bytes
  content: 0 bytes
  allocatedMiB: 0
  e2fsck rc: 0
CASE: 130MiB+8KiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/130MiB+8KiB_none.img
  requested: 136323072 bytes
  content: 0 bytes
  allocatedMiB: 0
  e2fsck rc: 0
CASE: 160MiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/160MiB_none.img
  requested: 167772160 bytes
  content: 0 bytes
  allocatedMiB: 0
  e2fsck rc: 0
CASE: 256MiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/256MiB_none.img
  requested: 268435456 bytes
  content: 0 bytes
  allocatedMiB: 0
  e2fsck rc: 0
CASE: 1GiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/1GiB_none.img
  requested: 1073741824 bytes
  content: 0 bytes
  allocatedMiB: 1
  e2fsck rc: 0
CASE: 4GiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/4GiB_none.img
  requested: 4294967296 bytes
  content: 0 bytes
  allocatedMiB: 7
  e2fsck rc: 0
CASE: 63x128MiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/63x128MiB_none.img
  requested: 8455716864 bytes
  content: 0 bytes
  allocatedMiB: 14
  e2fsck rc: 0
CASE: 63x128MiB+4KiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/63x128MiB+4KiB_none.img
  requested: 8455720960 bytes
  content: 0 bytes
  allocatedMiB: 14
  e2fsck rc: 0
CASE: 8GiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/8GiB_none.img
  requested: 8589934592 bytes
  content: 0 bytes
  allocatedMiB: 14
  e2fsck rc: 0
CASE: 16GiB_none
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/16GiB_none.img
  requested: 17179869184 bytes
  content: 0 bytes
  allocatedMiB: 29
  e2fsck rc: 0
CASE: 160MiB_10MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/160MiB_10MiB.img
  requested: 167772160 bytes
  content: 10485760 bytes
  allocatedMiB: 10
  e2fsck rc: 0
CASE: 128MiB_50MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/128MiB_50MiB.img
  requested: 134217728 bytes
  content: 52428800 bytes
  allocatedMiB: 50
  e2fsck rc: 0
CASE: 160MiB_120MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/160MiB_120MiB.img
  requested: 167772160 bytes
  content: 125829120 bytes
  allocatedMiB: 120
  e2fsck rc: 0
CASE: 160MiB_124MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/160MiB_124MiB.img
  requested: 167772160 bytes
  content: 130023424 bytes
  allocatedMiB: 124
  e2fsck rc: 0
CASE: 160MiB_126MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/160MiB_126MiB.img
  requested: 167772160 bytes
  content: 132120576 bytes
  allocatedMiB: 126
  e2fsck rc: 0
CASE: 256MiB_130MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/256MiB_130MiB.img
  requested: 268435456 bytes
  content: 136314880 bytes
  allocatedMiB: 130
  e2fsck rc: 0
CASE: 1GiB_200MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/1GiB_200MiB.img
  requested: 1073741824 bytes
  content: 209715200 bytes
  allocatedMiB: 201
  e2fsck rc: 0
CASE: 300MiB_260MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/300MiB_260MiB.img
  requested: 314572800 bytes
  content: 272629760 bytes
  allocatedMiB: 260
  e2fsck rc: 0
CASE: 4GiB_260MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/4GiB_260MiB.img
  requested: 4294967296 bytes
  content: 272629760 bytes
  allocatedMiB: 267
  e2fsck rc: 0
CASE: 63x128MiB_500MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/63x128MiB_500MiB.img
  requested: 8455716864 bytes
  content: 524288000 bytes
  allocatedMiB: 514
  e2fsck rc: 0
CASE: 8GiB_300MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/8GiB_300MiB.img
  requested: 8589934592 bytes
  content: 314572800 bytes
  allocatedMiB: 314
  e2fsck rc: 0
CASE: 16GiB_1000MiB
  image: /var/folders/y4/rkj6lthx2vg9ls8cbyyy8nb00000gn/T/ext4-e2fsck-matrix-92C353EF-EC99-40C4-8630-94E6E7CA3F46/16GiB_1000MiB.img
  requested: 17179869184 bytes
  content: 1048576000 bytes
  allocatedMiB: 1029
  e2fsck rc: 0
========== E2FSCK SUMMARY ==========
All 25 cases passed.

@dkovba please check according to your convience, thanks!

Open to making any changes needed or I missed for this current pull request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants